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ABSTRACT 


Holding a summer job is a rite of passage in American adolescence, a first rung towards 
adulthood and self-sufficiency. However, over the past decade, youth employment during the 
summer has decreased significantly. Summer youth employment has the potential to benefit high 
school students’ educational outcomes and employment trajectories, especially for low-income 
youth. Despite the potential importance of youth employment during summer, evidence of the 
impact of summer jobs on youth outcomes is limited to only a few studies. Our research 
examines summer youth employment, beginning with academic outcomes, by studying New 
York City’s Summer Youth Employment Program (SYEP). SYEP provides jobs to youth ages 
14-24, and due to high demand for summer jobs, allocates slots through a random lottery system, 
allowing for causal estimates of program impact. Our study uses student-level data from the 
New York City Department of Youth and Community Development (the SYEP administrating 
agency) and New York City Department of Education, encompassing approximately 300,000 
student SYEP applicants for the 2005-2009 program years. 

This paper examines the impact of SYEP on a wider range of academic performance 
outcomes, including test taking, passing rates and scores. It also attends to variation in these 
outcomes. Our findings suggest that SYEP has positive impacts on some student academic 
outcomes, and that these effects are heterogeneous. Future analyses will focus on examining 
program, student and school characteristics that might explain these variations. 
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INTRODUCTION 


Unemployment rates for youth jumped to historical highs after the recession of 2008 and 
have yet to recover. An important component of this jobs crisis is the lack of available summer 
jobs for high school students — especially low-income youth. Summer jobs for low-income 
youth represented a major component of The American Recovery and Reinvestment Act 
(ARRA), which provided $1.2 billion for youth employment opportunities and funded 345,000 
jobs during the summer of 2009 (Bellotti et al. 2010). However, these funds are no longer 
available, and many other publicly funded jobs have also experienced reductions in the number 
of youth they are able to employ. 

There is concern that the lack of employment opportunities for youth may hamper their 
career development, with lasting negative consequences. Prior research suggests that adolescent 
employment improves net worth and financial well-being as an adult (Painter, 2010; Ruhm, 
1995). Work experience may benefit youth, and high school students specifically, through 
multiple mechanisms. Beyond simply increasing income and work experience, jobs may provide 
opportunities for students to develop non-cognitive skills and “grit” (Heckman, 2000; Lillydahl, 
1990; Mortimer, 2003; Duckworth et al, 2007). Summer work is also considerably less likely 
than work during the school year to reduce the amount of time and effort allocated to educational 
pursuits. Further, an emerging body of rigorous research suggests that summer employment 
programs can lead to short-term decreases in violence and crime (Heller, 2014). Finally, Leos- 
Urbel (2014) examines impacts on school attendance, using data from a single cohort of 
applicants to New York City’s Summer Youth Employment Program (SYEP), finding significant 
positive effects on attendance and tantalizing, but mixed, evidence on the impact on standardized 
tests in English and Mathematics in high school. 
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This paper focuses on understanding the impact of summer youth employment on 
students’ academic achievement. We utilize a large data set including more than 190,000 
applications to New York City’s Summer Youth Employment Program (SYEP), over 4 years, 
from 2005- 2008, which we match to academic records from the New York City Department of 
Education (NYCDOE). Importantly, since the number of applicants substantially exceeds the 
number that can be served, positions are allocated through a random lottery, offering an unusual 
opportunity to derive estimates of the impact of the program. We use data on New York State’s 
“Regents” exams designed to assess performance in a variety of high school subjects including 
mathematics (including algebra, geometry and trigonometry), sciences (including biology, 
chemistry and physics), English, and History. Further, we examine the way in which the impact 
varies with repeated participation over multiple summers and explore differences in impacts 
across key subgroups. Future analysis will also examine the impact of SYEP on high school 
graduation outcomes, student reports of school experiences, attendance and persistence. In an 
effort to gain insight into the underlying mechanisms, future analyses will examine heterogeneity 
in treatment effects both by the type of summer job (e.g. summer camp versus retail), and the 
community based organization (CBO) placing and supervising students in jobs. Finally, using 
survey data collected by the NYCDOE, we will examine whether and how much SYEP 
influences students’ feelings and attitudes about their education and school. 

Preliminary findings suggest small positive effects of SYEP on the number of exams 
students take and, although impacts on scores are generally insignificant, there is a small positive 
effect on passing key high school exams. Further, we find that overall the improvements in test 
taking and passing increase with the number of years a student participates in SYEP - impacts 
are larger for second time participants and largest for those participating for the third time or 
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beyond. 


I. LITERATURE REVIEW AND CONCEPTUAL FRAMEWORK 

There are a multitude of mechanisms by which summer jobs may improve students’ 
academic outcomes as well as longer term economic success. In this section we draw on the 
existing literature related to these mechanisms, and highlight current gaps in our knowledge. 
In addition to improving students’ immediate financial well-being and strengthening their 
resumes, a considerable body of research suggests that work may improve “non-cognitive” 
skills such as responsibility, positive work habits, time management skills, determination, 
and self-confidence (Lillydahl, 1990; Mortimer, 2003; Heckman, 2000). Heckman (2000) 
argues that such non-cognitive skills and motivation are critical for success, and that these 
s ki lls can be improved at later ages. Also, recent work by Duckworth and colleagues (2007) 
suggest the importance of grit (defined as perseverance and passion for long-term goal) for 
success in school. 

Painter (2010), using the 1979 cohort of the National Longitudinal Survey of Youth, 
finds that students with longer high school work experience are associated with more wealth 
as adults. Moreover, students that work more intensely and over longer periods of time in 
high school have even higher lifetime wealth accumulation. Further, Painter suggests that 
employment during the summer might be particularly useful for students because it offers 
opportunities for full-time employment in jobs that may be otherwise unavailable to high 
school aged students. 

Another mechanism through which summer jobs may influence student success is 
through academic achievement and attainment. Much of the research examining the impact 
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of high school student employment on academic and other outcomes has been limited to 
work during the school year, focusing on the potential tradeoffs between the developmental 
and financial benefits of working and the possible crowding out of time devoted to academics 
(Rothstein, 2007; Sabia, 2009; Kalenkoski & Pabilonia, 2009). This research largely suggests 
that working a moderate number of hours (i.e., less than 20 hours per week) during the 
school year has either a small positive effect or no effect on outcomes such as school 
attendance, time spent on homework, and GPA, and that working a lot (i.e., more than 20 
hours per week) has negative effects on these outcomes (Lillydahl, 1990; Monahan, Lee, & 
Steinberg, 2011; Rothstein, 2007; Stem & Briggs, 2001). Most previous research, however, 
has explicitly excluded work experiences during the summer, when there is considerably less 
risk of detracting attention from school responsibilities (Painter, 2010; McNeal, 1997). 

Relatively few studies have assessed the effects of summer employment on student 
achievement directly. In their evaluation of the Summer Training and Education Program 
(STEP), Walker and Vilella-Velez (1992) find improved reading and mathematics test scores 
for academically behind 14 and 15-year-olds from poor urban families who participated in 
the program. STEP consisted of half-day summer jobs combined with half-days of academic 
coursework. The coursework consisted of specially designed remedial reading and 
mathematics curricula. In addition to higher test scores, participating students had better 
grade point averages, showed more knowledge about responsible sexual and social behavior, 
and had higher attendance rates than students from a control group. 

Leos-Urbel (2014) estimates the impact of New York City’s Summer Youth Employment 
Program (SYEP) on attendance for the 2007 cohort of students. He finds a significant 
increase in school attendance in the school year following SYEP participation, with larger 
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effects among students likely to be at greater risk of low attendance — students 16 years and 
older with low attendance rates in the previous year. His analysis of impacts on academic 
performance yielded some evidence that SYEP may increase the likelihood of attempting and 
passing statewide high school math and English exams. 

Finally, another mechanism by which summer jobs may improve student outcomes is by 
keeping participants occupied in positive and supervised activities. For instance, Anderson & 
Hughes (2009) find that unstructured time with peers is associated with greater delinquent 
behavior. Emerging research indicates that summer jobs programs may reduce violent 
behavior and criminal activity (Heller, 2013; Sum, Trubskyy & McHugh, 2013). 


II. POLICY CONTEXT 

New York City’s Summer Youth Employment program is designed to introduce and 
prepare youth for future careers, foster skills important for success in the labor market, and 
provide supplemental income to families. SYEP participants work in a variety of entry-level 
jobs at community-based organizations (CBOs), government agencies and private sector 
businesses; most common worksites include summer camps and day care, followed by social 
or community service agencies and retail. Participants are paid for up to 25 hours per week 
for up to six (or, in some years, seven) weeks at minimum wage, currently $7.25 per hour. In 
addition to work experience, ten percent of participant hours are dedicated to education and 
training on topics related to time management, financial literacy, workplace readiness and 
etiquette, career planning and finding employment. 

The New York City Department of Youth and Community Development (DYCD) 
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administers the program and contracts with a variety of CBOs to conduct intake and 
enrollment, as well as provide training and supervise job placement. All New York City 
(NYC) residents ages 14-24 are eligible to apply to SYEP. 1 To apply to the program, youth 
submit an application directly online or through a paper application and select a CBO service 
provider. Both applications are entered into the central SYEP data system. The system cross- 
checks across all service provider applications for duplication by matching the Social 
Security number and name of the applicant to ensure that each youth submits only one 
application for the program. Each complete application is randomly assigned an 
identification number. After the application deadline, DYCD assigns each service provider 
the number of SYEP slots that they are contracted to serve. DYCD then runs a lottery using 
the data system for each provider. The computerized system, using a random selection 
algorithm, selects applicants using the identification numbers for each provider according to 
the number of slots they have been allocated. The system sees each application as an ID 
number belonging to a provider and does not use any applicant information to determine their 
selection into the program, with the exception of those who have self-identified as having a 
disability. 

SYEP is funded through a combination of federal (including Workforce Investment Act, 
Community Services Block Grant and American Recovery and Reinvestment Act funds), 
state (state TANF and general funds), city (through a city tax levy) and private funds, and 
changes in the availability of program funding have dictated fluctuations in the number of 
participants served over time. Specifically, the increase in city and state funding after 2005 

1 SYEP also includes a few separate programs targeted at special populations, including one that serves only youth 
with disabilities through a separate lottery competition, a special program targeting vulnerable youth in foster care, 
court-involved or who are runaway/homeless youth that was added in 2009, and a school-year program funded 
through the Workforce Investment Act that does not use a lottery and guarantees admission. The results presented 
here focus on the larger general SYEP program and lottery only. 
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allowed DYCD to increase the number of participants from 33,739 in 2005 to 43,1 13 
participants by 2008. Expansion has not met demand, however, as the number of applications 
has almost doubled. SYEP received 69,328 applications in 2005; this number grew to 
103,189 in 2008. 


III. DATA AND SAMPLE 

Student-level data for this study come from two primary sources: SYEP files from the 
DYCD and NYCDOE administrative and survey data files. We have matched students from 
each of these files for the 2005-2008 program years, encompassing 195,289 student SYEP 
applications. Data include an indicator of SYEP lottery result, the CBO provider the student 
applied to, and, for those students who participate, the type of SYEP work placement, the 
specific worksite, and number of hours worked. Variables from NYCDOE files include 
student demographics, school attendance and information about standardized test-taking and 
performance. Student demographics include gender, race\ethnicity, English proficiency, 
participation in special education and ESL services, free and reduced price lunch eligibility, 
grade level, and age. 

Each student record includes information on test-taking and performance on New York 
State standardized tests in a variety of subjects, including English, Math A, Math B, 
Integrated Algebra and Geometry (which replaced Math A and B in later years), Global 
History, Earth Science, Biology, Physics and Chemistry. The Regents Examinations are a 


2 The data matching is a significant hurdle to have cleared, as DYCD and NYCDOE files do not contain a common 
ID. Data were matched on participant name and date of birth by an approved consultant with a match rate of 
approximately 70 percent or higher. However, unmatched records include an unknown number of students in 
private or parochial schools or schools outside of New York City, as well as non-students. The match rate only for 
NYCDOE students is likely considerably higher. Student files matched to a NYCDOE student ID number for a 
similar proportion of lottery winners and lottery losers. In order to maintain confidentiality, the data provided to 
investigators does not include participants’ names. 
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series of tests, aligned with New York State’s Learning Standards, designed and administered 
by the New York State (NYS) Department of Education, under the authority of the Board of 
Regents of the University of The State of New York and prepared by teacher examination 
committees and testing specialists. Examination scores range from 0-100%. Although the 
specific requirements change over time and students have some flexibility in choosing which 
exam to take, starting with students who entered 9th grade in 2001, earning a NYS high 
school diploma (“Regents’ Diploma”) requires passing a set of these exams including 
mathematics, English, Global History and Geography, US History and Government, and at 
least one science (e.g. Biology, Chemistry, Physics, Earth Science). More specifically, in 
order to graduate with a high school diploma, students must score 65 on any one math 
exam — usually Math A, 3 English, Global History and Geography and US History and 
Government, and one science exam. To earn and Advanced Regents Diploma, students pass 
an additional mathematics exam, Math B, 4 and one additional science (at least one life 
science and one physical science). Additionally, students entering 9th grade in 2007 and prior 
had the option of graduating with a “Local Diploma,” which required passing any one of five 
Regents exams with a score of at least 55. This option was gradually phased-out, 5 and the 
Local Diploma was not available for students entering 9th grade in 2008 and later. Regents 
exams in all subjects are offered in June each year, and a limited number of Regents are 
offered in January and August. There are no mandated grades in which students are eligible 
or required to take a specific exam, but they typically take the exam at the end of the related 

3 Last administered in January 2009 and replaced by Integrated Algebra beginning in June 2008 and Geometry 
beginning in June 2009. 

4 Last administered in June 2010 and replaced by Algebra 2 and Trigonometry in June 2009. 

5 Students entering grade 9 in 2005 were required to score 65 or above on two of the five required Regents exams 
and score 55 or above on the remaining three; 2006 9 th graders were required to score 65 or above on three of the 
five required exams, and 2007 9 th graders, 65 on four of the five required exams. 
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course. Because the graduation requirements reward passing but do not penalize failing, it is 
in a student’s best interest to take these exams earlier than later. The majority of students 
elect to take the exams in June at the end of the school year. 

Our analyses focus on the impact of SYEP participation on academic outcomes, 
including test-taking and test-performance. To assess student performance, we examine three 
test-related outcomes: test taking, passing, and the actual test score. We constructed an 
indicator variable for whether the student took the Regents exam in a particular subject and 
variables measuring performance in z-scores for each exam. 6 We also include indicator 
variables for whether the student passed the exam at three cut points: 55 (the score required 
for a Local Diploma available to a subset of students in our sample); 65 (required for a 
Regents diploma), and 75 (required on English and Math A for admission to CUNY four- 
year colleges). From these exam-specific indicators, we create seven measures to capture 
general performance on Regents exams: whether attempted any Regents exams in the school 
year following SYEP application and the total number of Regents exams attempted, whether 
passed any exams and total number of exams passed in that school year, the total number of 
exams passed with a score of 55 or above, the total number of exams passed with a score of 
65 or above, the total number of exams passed with a score of 75 or above, and the average 
(mean) score on all exams taken that year. 

Our sample includes all SYEP applicants who were NYC public school students, 
representing 178,441 applications to the program from 2005- 2008. Table 1 includes the 
number of SYEP applicants in each year as well as the number selected, and not selected, by 

6 z-scores are standardized to have a mean of zero and a standard deviation of one across all students taking that 
Regents exam in that particular year. 

7 We exclude duplicate observations for students who submit multiple SYEP applications within a year, and a 
subgroup who applied to vulnerable youth programs, WIA programs or programs that guaranteed summer jobs and 
did not use a lottery. 
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the lottery. Note that the number of applicants increased in each year, and that the percentage 
of applicants selected to participate decreased. Importantly, as illustrated in more detail in 
Table 6, some students applied to SYEP more than one time during this time frame, and these 
178,441 applications consist of 97,660 unique individuals. 

Our impact analyses focus on the 136,542 applications from students who were expected 
to attend high school in the school year following SYEP and thus likely to take Regents 

Q 

exams. Table 2 provides descriptive statistics for the outcomes of interest related to student 
Regents exam attempts and performance. Two-thirds of the sample attempted at least one 
Regents exam, with an average of 1.33 exams attempted. Roughly half of the sample passed 
at least one Regents exam, with students passing an average of 0.80 exams per year. The 
average z-score of -0.09 indicates that this sample performed slightly below the city average. 
Estimation Strategy 

This paper investigates the impact of SYEP on student academic success in the school 
year following SYEP participation, exploiting the random assignment of program 
participants. By comparing academic outcomes of students offered SYEP placements (the 
treatment group) to outcomes of students not offered placements (control group), we derive 
intent-to-treat estimates of the impact of SYEP. Since we also have data on whether the 
student actually participated in an SYEP program and the extent of this involvement, we can 
also estimate treatment effects of program participation. Our key outcomes are student level 
measures of attempting, passing, and performance (test scores) on the New York State 
standardized high school exams, including exams in Mathematics, English, History, and 
Science. Importantly, because SYEP participation is allocated via lottery, we are able to 

8 For these analyses we exclude students in grade 7 and lower and grade 12 in the school year prior to SYEP 
application, students in ungraded special education (who take Regents exams at low rates) and those who were not 
in New York City schools the school year after SYEP. 
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obtain causal estimates. If each SYEP lottery is random and there is no differential attrition, 


within any individual lottery, a simple comparison of means on the outcome of interest 
between those assigned to SYEP (treatment group) and those not (control group) provides 
unbiased estimates of the intent-to-treat (ITT) effect, where the treatment is a placement offer 
from SYEP. 9 In our analyses, the comparison group is the set of students who applied to 
SYEP in a particular summer, but who were not offered a placement. These students should 
be otherwise similar to the students in the treated group across all dimensions and, most 
importantly, similar in unobserved characteristics, such as motivation and other non- 
cognitive attributes. 10 

We estimate the impact of SYEP using equation (1). Here Y is the outcome of interest for 
student i in year t such that: 

1 Y it = Po + P ± SYEP it + P 2 ST it + 8 cbo + y t + Pit 
where SYEP takes a value of 1 if the student i was made the offer to participate in SYEP and 
0 if he/she was not. ST is a vector of student characteristics that may influence student 
performance, such as gender, race/ethnicity, free and reduced price lunch eligibility, limited 
English proficiency, special education status, ESL status, and grade. 5 are provider fixed 
effects which allow a different intercept for each of the community-based organization 
(CBOs) providers to control for differences in the selection rates and potential differences in 
applicants across CBOs. In the model y are cohort fixed effects, based on a student’s first 
year applying to SYEP and grade in the school year prior to applying to SYEP, and p is an 


9 See appendix for verification that the lottery is in fact random. 

10 In order to evaluate the possibility that admission to the program is not random, we estimated the effect of 
winning the lottery on each pre-existing student characteristic. Specifically, for each observed characteristic, we test 
whether winning the lottery is uncorrelated by regressing each characteristic on a full set of indicators for CBOs and 
indicators for receiving treatment interacted with CBO (results are presented in the Appendix, Table 1 A). Results 
from a joint cross-equation cross-model F-test that all treatment-by-CBO interaction coefficients are equal to zero 
suggest that the lottery selection process was random (p>. 10). 
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error term with usual properties. Standard errors are clustered at the CBO level. In this 
model, is the primary parameter of interest and captures the impact of assignment to 
SYEP. 

We use seven key measures of academic success related to test-taking and test- 
performance, including passing and z-scores. Our initial models examine performance across 
all Regents exams in the school year following SYEP application. These outcomes all 
capture important measures of educational progress, effort, and ultimately success. In 
addition to being a necessary pre-condition for graduation, attempting the Regents may also 
be a signal of academic interest, engagement, and effort. If participation in SYEP encourages 
students to increase their school effort, they may elect to take more Regents exams than the 
minimum required for graduation, potentially improving their preparation for college. 

Further, to the degree that participation in SYEP encourages academic effort, there may be an 
improvement in student performance on these exams - both in terms of passing and the 
actual score - if students are more attentive in class or spend more time studying and 
preparing for exams. 

For models where the outcome is an indicator variable (test-taking and test passing), we 
estimate this equation using linear probability models. Note that although covariates are not 
necessary to derive unbiased impact estimates when treatment is randomly assigned, model 
(1) includes student characteristics to improve the precision of our estimates (see Bloom, 
2006 ). 

Impact of the second and third year 

An important feature of the SYEP program is that each lottery is unrelated to lotteries in 
the previous and subsequent years. Students who participate in SYEP in year t, therefore, are 
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eligible to apply again in t+1. Thus, there are a group of students who participate in t and 
apply again in t+1. Among this group of previous participants, a randomly assigned group 
will be offered a placement in t+1. Thus, we can derive an estimate of the causal effect of a 
second year of SYEP, conditional on having participated the year before. 

We may imagine that the impact of SYEP may vary for those who had applied (and 
participated) in previous years. First, for those who apply, win the lottery, and participate in 
multiple years, there may be a dosage effect, in which participating for more than one 
summer is important to fully realize the effects of SYEP. Conversely, although the SYEP 
lottery does not take into account whether a student had applied or participated before, the 
decision to apply for multiple years itself is not random, and it may be that the types of 
students who apply for multiple years benefit the most. For example, students who do not 
have access to alternate activities or means of finding employment, or more motivated 
students might be more likely to apply for an additional year. Additionally, the decision to 
apply to SYEP for a second or third year may be due to a positive work experience the first 
year of SYEP. In other words, a finding of positive or stronger effects in later years could be 
because there are increasing returns to participants for each year they participate, or simply 
because the estimates are for the types of students who are most likely to benefit from SYEP 
in any year. 

To estimate the impact of the second year of SYEP, we first re-estimate model (1) but 
limit the sample based on year of application. First, we limit the sample to include first-time 
applicants only to estimate the impact of one year. Next, we estimate the impact of winning 
the SYEP lottery for a second time, limiting the sample to students who had participated in 
SYEP in a prior year (i.e. applied, won the lottery, and worked in the program) then applied 
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again. We employ the same strategy for students who participated for two years or more to 
estimate the impact of winning the lottery for a third year, limiting the sample to the set of 
students that applied three times, after having participated in the program in the two years 
prior. Finally, we re-estimate the effect of winning the lottery in the first year, breaking out 
effects by whether students applied only once or applied again in future years, and then 
similarly estimate the effect of winning the lottery in the second year by whether the students 
apply again — for a third time — in future years. Although these last two sets of estimates are 
correlational and not causal they provide us with a better understanding of the extent to 
which the effects of SYEP vary across the three groups of students in order to begin to 
disentangle selection into a second year from the effect of multiple years. 

IV. RESULTS 

Table 3 presents results for models estimating the impact of SYEP on Regents exam 
outcomes in the following school year. All models include demographic controls including 
free and reduced lunch eligibility, race/ethnicity, gender, special education status and English 
language learner status, as well as CBO, grade, and cohort fixed effects. The results in 
column 1 indicate that winning the SYEP lottery has a small significant effect on whether 
students attempt at least one Regents exam. Column 2 indicates a small statistically 
significant effect of winning the SYEP lottery on the number of exams attempted - an 
increase of 0.02 exams. Columns 3 and 4 suggest a small significant increase in ever passing 
any Regents exam, as well as in the number of exams passed, for SYEP lottery winners. 
Column 5 finds a small significant increase in the number of exams with a score of 55 or 
higher, and column 6 indicates a small marginally significant effect on the number of exams 
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with a score of 75 or higher, which constitute a high level of achievement. Finally, column 7 
indicates no impact on mean standardized scores on these exams. Taken together, these 
results suggest that SYEP has a small positive effect on taking and passing Regents exams. 

Although Table 3 provides estimates of SYEP impacts for students who won an SYEP 
lottery, regardless of whether they actually enrolled or worked in the program, Table 4 
indicates that only approximately three-quarters of the applicants selected by lottery actually 
participated in the program. Because not all lottery winners work, in addition to estimating 
the intent-to-treat (ITT) impact we also estimate the impact of participating in SYEP — i.e. 
the treatment-on-the-treated (TOT) effect — using a quasi-experimental approach. 
Specifically, we use instrumental variable analysis in which winning the SYEP lottery 
indicator serves as an instrument for participation. Table 5 displays estimates of the TOT 
impact on Regents outcomes. Our results suggest that impacts of participating in SYEP are 
small and positive, and these effects are approximately 1.3 times greater than the ITT 
estimates. 

The results presented in Tables 3 and 5 represent the impact of SYEP on Regents 
outcomes in the following school year, irrespective of the number of times that students 
applied to or participated in SYEP. Table 6 provides information regarding patterns in 
application to and selection by the SYEP lottery over the four-year study period. While 
roughly two-thirds of the sample applied in only one year, one-third applied more than once, 
with 25 percent applying twice, 7 percent applying three times, and 1 percent applying four 
times. Among these applicants, 39 percent never won the SYEP lottery, 49 percent won 
once, 1 1 percent twice, and 2 percent three times. 

The next set of tables present results for models estimating the impact of SYEP on 
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Regents outcomes, taking into account the number of years of participation in the program. 
Results from these models suggest that the impacts of SYEP are substantively larger for 
students who participate in SYEP for a second and third year. 

The results presented in Table 7 indicate no significant effect of winning the lottery for 
all first-time applicants, with one exception — a small marginally significant increase in the 
number of exams with a score of 55 or higher. In contrast, Table 8 suggests that for students 
that had participated in SYEP in a prior year, winning the lottery for a second year results in 
significant increases in the likelihood that they ever attempt a Regents exam and ever pass an 
exam with a score of 65, as well as significant increases in the number of exams attempted 
and the number passed with a score of 55 and 75. Table 9 presents models estimating the 
impact of winning the SYEP lottery for students that had participated in two prior years. For 
these students, the coefficients for ever attempting and the number of exams attempted are 
positive but not significant, while those for ever passing with a score of 65, the number of 
exams passed at the 55 and 65 thresholds, as well as the actual standardized test score (z- 
score) are significant, and larger than for those who won the lottery after having participated 
in only one prior year. 

Table 10 presents models estimating the effects of SYEP for first- year applicants, 
providing separate estimates by whether the applicants apply again in the future or not. 
Results suggest that the effect of winning the lottery differs depending on whether students 
apply again in future years: winning the lottery in the first year of application has small 
positive effects on students who apply again in the future. This finding holds true across all 
outcomes. Winning the lottery in the first year appears to have very small, negative effects on 
students who do not apply for a second time or more. Similarly, results presented in Table 1 1 
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suggest that among students who apply for a second time, effects of winning the lottery 
depend on whether students apply again in the future, for a third time. Winning the lottery 
has positive effects on the number of exams attempted and the number of exams passed at the 
55 threshold for students who apply for a third time. Results suggest winning the lottery has 
no effect on second-time applicants who do not apply again. 


IV. DISCUSSION AND CONCLUSION 

Taken together, our estimates suggest that participation in SYEP has, on average, a very 
small positive effect on taking and passing the standardized tests administered by New York 
State to measure progress in high school subjects. These average effects mask the significant 
difference in the impact of participating the first time and participating the second (or third) 
time. Disentangling these effects reveals, in fact, little or no effect of a single year of 
participation, but somewhat larger, positive effects for the second and third year of 
participation. We do, however, find small positive effects for those first year participants who 
will ultimately participate multiple times. Further analysis will aim to shed let on these 
findings, for instance by examining the differences in characteristics of students who apply to 
SYEP for multiple years, compared to those who only apply one time, as well as the 
characteristics of their first-year jobs or CBOs. 

While much additional work remains to be done, the results offer evidence that SYEP 
improves educational outcomes that have proven stubbornly resistant to interventions. As an 
example, New York City’s Conditional Cash Transfer program offered high school students 
$600 incentive for each Regents exam passed-up to five-but yielded no significant effect 
(Riccio et al., 2013). Interestingly, larger effects were found for students who were deemed 
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proficient in English Language Arts and Mathematics at the time they enrolled in high 
school, suggesting this is a subgroup worthy of future investigation in the SYEP analysis. 
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Table 1. Sample by lottery outcome, 2005-2008 



Lottery 

winners 

Lottery 

losers 

Total 

2005 

20,237 

11,175 

31,412 

2006 

23,158 

14,398 

37,556 

2007 

27,049 

24,404 

51,453 

2008 

28,747 

29,273 

58,020 

Total 

99,191 

79,250 

178,441 


Note: Sample excludes duplicate observations for students who submit multiple SYEP applications and a subgroup who 
applied to vulnerable youth programs or programs that guaranteed summer jobs and did not use a lottery. 
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Table 2. Regents Exam Outcomes in School Year Following SYEP, 2005-2008 


Variable 

Mean 

Std. Dev. 

Min 

Max 

Attempt any regents exams 

0.66 

0.47 

0 

1 

Number of regents attempted 

1.33 

1.24 

0 

7 

Pass any exams (65+) 

0.48 

0.50 

0 

1 

Number of exams passed (65+) 

0.80 

1.00 

0 

7 

Number of exams with score 55+ 

1.00 

1.09 

0 

7 

Number of exams with score 75+ 

0.40 

0.76 

0 

5 

Average z-score 

-0.09 

0.84 

-6.06 

2.37 
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Table 3: Baseline models, 2005-2008 SYEP applicants 



Ever attempt 
(1) 

Number attempts 
(2) 

Ever pass 65 
(3) 

Number pass 65 
(4) 

Number pass 55 
(5) 

Number pass 75 
(6) 

z score 
(7) 


Win lottery 

0.008*** 

0.022*** 

0.010*** 

0.019*** 

0.026*** 

0.007* 

0.007 


(0.002) 

(0.006) 

(0.002) 

(0.005) 

(0.005) 

(0.004) 

(0.005) 

Student Chars 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Cohort FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Provider FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

136,542 

136,542 

136,542 

136,542 

136,542 

136,542 

90,170 

R-squared 

0.147 

0.234 

0.152 

0.202 

0.204 

0.188 

0.183 


Notes: Robust standard errors in parentheses; *** p<0.01, ** p<0.05, * p<0.1. A cohort is the year and grade of first application to SYEP. Sample includes all 
students expected to be in high school following SYEP and excludes duplicate observations for students who submit multiple SYEP applications, are in ungraded 
special education, and a subgroup who applied to vulnerable youth programs or programs that guaranteed summer jobs and did not use a lottery. Regents 
exams include English, Math A, Math B, Geometry, Integrated Algebra, Living Environment/Biology, Physical Setting/Earth Science, Physical Setting/Physics, 
Physical Setting/Chemistry and Global History and Geography. The mean Regents exam score is calculated as the mean of a student's z-scores on all Regents 
exams taken the school year following having applied to SYEP; Regents scores are standardized (z-scored) within test and year of administration. Demographic 
controls include free and reduced lunch eligibility, race/ethnicity, gender, special education status, English language learner status, ESL status, and grade. 
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Table 4 . SYEP take-up rates, 2005-2008 


% of lottery winners 
that worked 


Number of winners 


2005 81.8 

2006 83.6 

2007 73.4 

2008 74.5 

Total 77.9 


15,846 

17,522 

20,043 

20,555 

73,966 


Notes: Sample includes all students expected to be in high school 
following SYEP, and excludes duplicate observations for students 
who submit multiple SYEP applications, are in ungraded special 
education and a subgroup who applied to vulnerable youth 
programs or programs that guaranteed summer jobs and did not 
use a lottery. 
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Table 5: Regents outcomes in school year following SYEP application, treatment-on-treated (TOT) models, 2005-2008 SYEP 
applicants 



Ever attempt 
(1) 

Number attempts 
(2) 

Ever pass 65 
(3) 

Number pass 65 
(4) 

Number pass 55 
(5) 

Number pass 75 
(6) 

z score 
(7) 


Win lottery 

0.010*** 

0.029*** 

0.013*** 

0.024*** 

0.034*** 

0.010* 

0.009 


(0.003) 

(0.007) 

(0.003) 

(0.006) 

(0.006) 

(0.005) 

(0.006) 

Student Chars 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Cohort FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Provider FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

136,542 

136,542 

136,542 

136,542 

136,542 

136,542 

90,170 

R-squared 

0.081 

0.120 

0.084 

0.100 

0.105 

0.084 

0.094 


Notes: Robust standard errors in parentheses; *** p<0.01, ** p<0.05, * p<0.1. A cohort is the year and grade of first application to SYEP. Sample includes all 
students expected to be in high school following SYEP, and excludes duplicate observations for students who submit multiple SYEP applications, are in 
ungraded special education and a subgroup who applied to vulnerable youth programs or programs that guaranteed summer jobs and did not use a lottery. 
Regents exams include English, Math A, Math B, Geometry, Integrated Algebra, Biology, Earth Science, Physics, Chemistry and Global History and Geography. 
Demographic controls include free and reduced lunch eligibility, race/ethnicity, gender, special education status, English language learner status, ESL status, 
and grade. 
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Table 6: Cross-tabulation of number of applications and number of wins, 2005-2008 


Number of 
applications 


Number of wins 



0 

1 

2 

3 

4 

Total 

1 

32,081 

34,603 

0 

0 

0 

66,684 

2 

5,072 

10,968 

7,700 

0 

0 

23,740 

3 

572 

1,984 

2,557 

1,369 

0 

6,482 

4 

34 

109 

265 

233 

113 

754 

Total 

37,759 

47,664 

10,522 

1,602 

113 

97,660 


Note: Sample includes all students expected to be in high school following SYEP and excludes duplicate observations for students who submit multiple SYEP 
applications, are in ungraded special education, and a subgroup who applied to vulnerable youth programs or programs that guaranteed summer jobs and did 
not use a lottery. 
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Table 7: Regression Results, first time SYEP applicants, 2006-2008 



Ever attempt 
(1) 

Number attempts 
(2) 

Ever pass 65 
(3) 

Number pass 65 
(4) 

Number pass 55 
(5) 

Number pass 75 
(6) 

z score 
(7) 

Win 

-0.001 

0.010 

0.002 

0.008 

0.012* 

0.002 

0.003 


(0.003) 

(0.008) 

(0.003) 

(0.007) 

(0.007) 

(0.007) 

(0.007) 

Student Chars 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Cohort FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Provider FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

68,563 

68,563 

68,563 

68,563 

68,563 

68,563 

46,713 

R-squared 

0.146 

0.248 

0.150 

0.214 

0.216 

0.201 

0.193 


Notes: Standard errors clustered by CBO in parentheses; *** p<0.01, ** p<0.05, * p<0.1. A cohort is defined by the year and grade of first application to SYEP. 
Sample is limited to SYEP participants applying for the first time and includes all students expected to be in high school following SYEP and excludes duplicate 
observations for students who submit multiple SYEP applications, are in ungraded special education, and a subgroup who applied to vulnerable youth 


programs or programs that guaranteed summer jobs and did not use a lottery. Regents exams include English, Math A, Math B, Geometry, Integrated Algebra, 
Living Environment/Biology, Physical Setting/Earth Science, Physical Setting/Physics, Physical Setting/Chemistry and Global History and Geography. The mean 
Regents exam score is calculated as the mean of a student's z-scores on all Regents exams taken the school year following having applied to SYEP; Regents 
scores are standardized (z-scored) within test and year of administration. Student characteristics include free and reduced lunch eligibility, race/ethnicity, 
gender, special education status, English language learner status, ESL status, and grade. 
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Table 8: Regression results, second-time SYEP applicants, previous participants, 2006-2008 



Ever attempt 
(1) 

Number attempts 
(2) 

Ever pass 65 
(3) 

Number pass 65 
(4) 

Number pass 55 
(5) 

Number pass 75 
(6) 

z score 
(7) 

Win 

0.019** 

0.035* 

0.017** 

0.018 

0.039** 

0.017* 

0.003 


(0.007) 

(0.019) 

(0.008) 

(0.016) 

(0.018) 

(0.009) 

(0.016) 

Student Chars 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Cohort FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Provider FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

18,071 

18,071 

18,071 

18,071 

18,071 

18,071 

12,285 

R-squared 

0.187 

0.275 

0.195 

0.242 

0.235 

0.215 

0.189 


Notes: Standard errors clustered by CBO in parentheses; *** p<0.01, ** p<0.05, * p<0.1. A cohort is defined by the year and grade of first application to SYEP. 
Sample is limited to SYEP participants applying for the second time and includes all students expected to be in high school following SYEP and excludes 
duplicate observations for students who submit multiple SYEP applications, are in ungraded special education, and a subgroup who applied to vulnerable youth 
programs or programs that guaranteed summer jobs and did not use a lottery. Regents exams include English, Math A, Math B, Geometry, Integrated Algebra, 
Living Environment/Biology, Physical Setting/Earth Science, Physical Setting/Physics, Physical Setting/Chemistry and Global History and Geography. The mean 
Regents exam score is calculated as the mean of a student's z-scores on all Regents exams taken the school year following having applied to SYEP; Regents 
scores are standardized (z-scored) within test and year of administration. Student characteristics include free and reduced lunch eligibility, race/ethnicity, 
gender, special education status, English language learner status, ESL status, and grade. 
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Table 9: Regression results, third-time SYEP applicants, two time participants, 2006-2008 



Ever attempt 
(1) 

Number attempts 
(2) 

Ever pass 65 
(3) 

Number pass 65 
(4) 

Number pass 55 
(5) 

Number pass 75 
(6) 

z score 
(7) 

Win 

0.015 

0.038 

0.031** 

0.057** 

0.061* 

0.008 

0.071** 


(0.016) 

(0.045) 

(0.013) 

(0.026) 

(0.032) 

(0.019) 

(0.033) 

Student Chars 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Cohort FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Provider FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

3,178 

3,178 

3,178 

3,178 

3,178 

3,178 

1,893 

R-squared 

0.213 

0.318 

0.238 

0.307 

0.296 

0.254 

0.211 


Notes: Standard errors clustered by CBO in parentheses; *** p<0.01, ** p<0.05, * p<0.1. A cohort is defined by the year and grade of first application to SYEP. 
Sample is limited to SYEP participants applying for the third time and includes all students expected to be in high school following SYEP and excludes duplicate 
observations for students who submit multiple SYEP applications, are in ungraded special education, and a subgroup who applied to vulnerable youth 


programs or programs that guaranteed summer jobs and did not use a lottery. Regents exams include English, Math A, Math B, Geometry, Integrated Algebra, 
Living Environment/Biology, Physical Setting/Earth Science, Physical Setting/Physics, Physical Setting/Chemistry and Global History and Geography. The mean 
Regents exam score is calculated as the mean of a student's z-scores on all Regents exams taken the school year following having applied to SYEP; Regents 
scores are standardized (z-scored) within test and year of administration. Student characteristics include free and reduced lunch eligibility, race/ethnicity, 
gender, special education status, English language learner status, ESL status, and grade. 
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Table 10: Regression results, first time SYEP applicants by future application, 2006-2008 



Ever attempt 

Num attempts 

Ever pass 65 

Num pass 65 

Num pass 55 

Num pass 75 

z score 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

Win x apply again 

0.023** 

0.045*** 

0.026*** 

0.042*** 

0.042*** 

0.029** 

0.010 


(0.009) 

(0.017) 

(0.009) 

(0.015) 

(0.015) 

(0.012) 

(0.014) 

Win 

-0.014*** 

-0.015* 

-0.010*** 

-0.013* 

-0.010 

-0.010 

-0.003 


(0.004) 

(0.008) 

(0.004) 

(0.007) 

(0.007) 

(0.007) 

(0.008) 

Will apply again 

0.151*** 

0.303*** 

0.125*** 

0.217*** 

0.263*** 

0.096*** 

0.064*** 


(0.007) 

(0.013) 

(0.007) 

(0.012) 

(0.011) 

(0.010) 

(0.017) 

Student Chars 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Cohort FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Provider FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

68,563 

68,563 

68,563 

68,563 

68,563 

68,563 

46,713 

R-squared 

0.162 

0.257 

0.160 

0.221 

0.226 

0.204 

0.194 

Notes: Standard errors clustered by CBO in parentheses; *** p<0.01, ** p<0.05, * 

p<0.1. A cohort is defined by the year and grade of first application to SYEP. 

Sample is limited to SYEP participants applying for the first time and includes all students expected to be in high school following SYEP and excludes duplicate 
observations for students who submit multiple SYEP applications, are in ungraded special education, and a subgroup who applied to vulnerable youth 
programs or programs that guaranteed summer jobs and did not use a lottery. Regents exams include English, Math A, Math B, Geometry, Integrated Algebra, 


Living Environment/Biology, Physical Setting/Earth Science, Physical Setting/Physics, Physical Setting/Chemistry and Global History and Geography. The mean 
Regents exam score is calculated as the mean of a student's z-scores on all Regents exams taken the school year following having applied to SYEP; Regents 
scores are standardized (z-scored) within test and year of administration. Student characteristics include free and reduced lunch eligibility, race/ethnicity, 
gender, special education status, English language learner status, ESL status, and grade. 
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Table 11: Regression results, second time SYEP applicants by future applications, 2006-2008 



Ever attempt 
(1) 

Num attempts 
(2) 

Ever pass 65 
(3) 

Num pass 65 
(4) 

Num pass 55 
(5) 

Num pass 75 
(6) 

z score 
(7) 

Win x apply again 

0.020 

0.102*** 

0.020 

0.035 

0.072** 

0.016 

-0.017 


(0.016) 

(0.033) 

(0.016) 

(0.032) 

(0.035) 

(0.025) 

(0.026) 

Win 

0.007 

-0.006 

0.006 

-0.002 

0.007 

0.008 

0.005 


(0.009) 

(0.021) 

(0.010) 

(0.018) 

(0.021) 

(0.011) 

(0.018) 

Will apply again 

0.147*** 

0.268*** 

0.136*** 

0.246*** 

0.279*** 

0.133*** 

0.089*** 


(0.013) 

(0.029) 

(0.014) 

(0.030) 

(0.029) 

(0.023) 

(0.029) 

Student Chars 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Cohort FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Provider FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

18,071 

18,071 

18,071 

18,071 

18,071 

18,071 

12,285 

R-squared 

0.203 

0.284 

0.207 

0.252 

0.248 

0.220 

0.191 


Notes: Standard errors clustered by CBO in parentheses; *** p<0.01, ** p<0.05 / * p<0.1. A cohort is defined by the year and grade of first application to SYEP. 
Sample is limited to SYEP participants applying for the second time and includes all students expected to be in high school following SYEP and excludes 


duplicate observations for students who submit multiple SYEP applications, are in ungraded special education, and a subgroup who applied to vulnerable youth 
programs or programs that guaranteed summer jobs and did not use a lottery. Regents exams include English, Math A, Math B, Geometry, Integrated Algebra, 
Living Environment/Biology, Physical Setting/Earth Science, Physical Setting/Physics, Physical Setting/Chemistry and Global History and Geography. The mean 
Regents exam score is calculated as the mean of a student's z-scores on all Regents exams taken the school year following having applied to SYEP; Regents 
scores are standardized (z-scored) within test and year of administration. Student characteristics include free and reduced lunch eligibility, race/ethnicity, 
gender, special education status, English language learner status, ESL status, and grade. 
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APPENDIX 


Table Al: Lottery randomization results 

2005 2006 2007 2008 

F 1.01 1.06 1.02 1.06 

Prob > F 0.4221 0.1422 0.3316 0.1018 


Notes: We are testing if the treatment is uncorrelated with each observed characteristic for each CBO. 
We implement the test of randomization by regressing each characteristic on a full set of indicators 
for the CBO and indicators for receiving treatment interacted with CBO attendance. 


36 


Table A2: Attrition in year following application to SYEP, Grade 8-11 and alternative 


program, 2005-08 
Grade 

% attrition, 
selected students 

% attrition, not 
selected 

% attrition, all SYEP 
students 

N 

Grade 8 

5.9 

5.6 

5.8 

21,075 

Grade 9 

7.0 

7.5 

7.2 

51,264 

Grade 10 

5.2 

5.5 

5.4 

42,921 

Grade 11 

6.7 

6.8 

6.8 

23,831 

Alternative 

49.1 

50.7 

49.9 

1,128 

program 
Not in DOE data 

83.6 

65.6 

78.2 

6,533 

Total 

10.9 

8.6 

9.9 

146,674 


Notes: Attrition is characterized as not appearing in DOE administrative data in the year following the SYEP lottery. 
Students in alternative grades are those students that are enrolled in GED completion programs. Students missing 
grades are students that are not in the DOE the school year of the lottery. Sample excludes students in lotteries 
that explicitly serve WIA and vulnerable youth, and students that apply to lotteries with perfect selection. 
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Table A3: Impact of selection on attrition, by grade 



All grades 
(1) 

Grade 8 
(2) 

Grade 9 
(3) 

Grade 10 
(4) 

Grade 11 
(5) 

Alt program 
(6) 

Missing grade 
(7) 

Win lottery 

0.002 

0.005 

-0.003 

0.002 

0.002 

0.045 

0.020 


(0.001) 

(0.003) 

(0.002) 

(0.002) 

(0.004) 

(0.027) 

(0.016) 

Grade FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Year FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

CBO FX 

Y 

Y 

Y 

Y 

Y 

Y 

Y 

Observations 

146,752 

21,075 

51,264 

42,921 

23,831 

1,128 

6,533 

R-squared 

0.275 

0.013 

0.014 

0.011 

0.010 

0.137 

0.258 


Notes: Standard errors robust to clustering at the CBO level shown in parentheses (*** p<0.01, ** p<0.05, * 
p<0.1). Attrition is characterized as not appearing in DOE administrative data in the year following the SYEP lottery. 
Students in alternative grades are those students that are enrolled in GED completion programs. Students missing 
grades are students that are not in the DOE the school year of the lottery. Sample excludes students in lotteries 
that explicitly serve WIA and vulnerable youth, and students that apply to lotteries with perfect selection. 
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